Institut für Informatik Neuroinformatics Group Reinforcement Learning with Recurrent Neural Networks

Controlling a high-dimensional dynamical system with continuous state and action spaces in a partially unknown environment like a gas turbine is a challenging problem. So far often hard coded rules based on experts’ knowledge and experience are used. Machine learning techniques, which comprise the field of reinforcement learning, are generally only applied to sub-problems. A reason for this is that most standard reinforcement learning approaches still fail to produce satisfactory results in those complex environments. Besides, they are rarely data-efficient, a fact which is crucial for most real-world applications, where the available amount of data is limited. In this thesis recurrent neural reinforcement learning approaches to identify and control dynamical systems in discrete time are presented. They form a novel connection between recurrent neural networks (RNN) and reinforcement learning (RL) techniques. Thereby, instead of focusing on algorithms, neural network architectures are put in the foreground. RNN are used as they allow for the identification of dynamical systems in form of high-dimensional, non-linear state space models. Also, they have shown to be very data-efficient. In addition, a proof is given for their universal approximation capability of open dynamical systems. Moreover, it is pointed out that they are, in contrast to an often cited statement, well able to capture long-term dependencies. As a first step towards reinforcement learning, it is shown that RNN can well map and reconstruct (partially observable) Markov decision processes. In doing so, the resulting inner state of the network can be used as a basis for standard RL algorithms. This so-called hybrid RNN approach is rather simple but showed good results for a couple of applications. The further developed recurrent control neural network combines system identification and determination of an optimal policy in one network. It does not only learn from data but also integrates prior knowledge into the modelling in form of architectural concepts. Furthermore, in contrast to most RL methods, it determines the optimal policy directly without making use of a value function. This distinguishes the approach also from other works on reinforcement learning with recurrent networks. The methods are tested on several standard benchmark problems. In addition, they are applied to different kinds of gas turbine simulations of industrial scale.